Functional image archiving

ABSTRACT

Some implementations related to archiving of functional images. In some implementations, a method includes accessing images and determining one or more functional labels corresponding to each of the images and one or more confidence scores corresponding to the functional labels. A functional image score is determined for each of the images based on the functional labels having a corresponding confidence score that meets a respective threshold for the functional labels. In response to determining that the functional image score meets a functional image score threshold, a functional image signal is provided that indicates that one or more of the images that meet the functional image score threshold are functional images. The functional images are determined to be archived, and are archived by associating an archive attribute with the functional images such that functional images having the archive attribute are excluded from display in views of the images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/230,788, filed Dec. 21, 2018 and titled FUNCTIONAL IMAGE ARCHIVING,which claims priority to U.S. Provisional Patent Application No.62/609,522, filed Dec. 22, 2017 and titled FUNCTIONAL IMAGE ARCHIVING,the contents of both of which are incorporated by reference herein intheir entirety.

BACKGROUND

Digital albums often include images of a utilitarian or functionalnature such as images of documents, receipts, credit cards, membershipcards, Wi-Fi passwords, memes, etc. When viewing an album, or generatinga composition of images, functional images may tend to interrupt auser's flow of viewing, reminiscing or sharing other non-functionalimages (e.g., family photos, scenic photos, event photos, etc.).

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

SUMMARY

Some implementations are generally related to digital image management,and in particular, to automatically archiving or providing automaticsuggestions to archive functional images.

In some implementations, a computer-implemented method includesaccessing a plurality of images and determining one or more functionallabels corresponding to each of the plurality of images and one or moreconfidence scores corresponding to the one or more functional labels.The method includes determining a functional image score for each of theplurality of images based on the one or more functional labels having acorresponding confidence score that meets a respective threshold for theone or more functional labels. The method includes, in response todetermining that the functional image score meets a functional imagescore threshold, providing a functional image signal that indicates thatone or more of the plurality of images that meet the functional imagescore threshold are one or more functional images. The method includesdetermining that the one or more functional images are to be archived,and causing the one or more functional images to be archived byassociating an archive attribute with the one or more functional imagessuch that the one or more functional images having the archive attributeare excluded from display in views of the plurality of images.

Various implementations and examples of the method are described, whichcan be combined in various implementations as well. For example, in someimplementations, the method further comprises determining whether one ormore images of the plurality of images are included in one or morefunctional image categories. In some implementations, determining thatthe one or more functional images are to be archived is further based ondetermining that the one or more functional images are included in atleast one of the one or more functional image categories. In someimplementations, determining that the one or more functional images areto be archived is performed automatically when the one or morefunctional image categories correspond to predetermined archivingcategories and the functional image score meets a respective archivingthreshold for the one or more functional image categories. In someexamples, the one or more functional image categories include at leastone of: receipt, document, newspaper, note, product label, restaurantmenu, identification, screenshot, or business card.

In some implementations, determining one or more functional labelscorresponding to each of the plurality of images and one or moreconfidence scores corresponding to the one or more functional labelsincludes programmatically analyzing one or more pixels of each of theplurality of images to detect content of the respective images,identifying one or more labels corresponding to each of the plurality ofimages based on the content of the image, determining whether one ormore functional labels are present within the one or more labels; and,if one or more functional labels are present, selecting the one or morefunctional labels and the one or more corresponding confidence scores toassociate with one or more respective images of the plurality of images.In some examples, determining whether one or more functional labels arepresent within the one or more labels includes determining whether theone or more functional labels match predetermined labels associated witha functional image designation.

In some implementations, the method further comprises determining arecommendation to archive the functional images, where therecommendation is based on respective functional image scores; andcausing output of the recommendation by a device, where determining thatthe one or more functional images are to be archived is based on userinput received at the device in response to the providing therecommendation. In some examples, causing output of the recommendationincludes displaying a user interface card and a visual representation ofat least a subset of the one or more functional images. In someexamples, the recommendation includes at least one category associatedwith at least one of the one or more functional images, where the atleast one category is based on the one or more functional labels for theone or more functional images. In some examples, the recommendation isprovided by surfacing a suggestion chip element in a user interface thatdisplays the image.

In some implementations, the method is performed at one of: a time theplurality of images is obtained; or a time after a time period elapsesafter the plurality of images was obtained. In some implementations,determining the functional image score includes programmaticallyanalyzing one or more of image content data associated with one or moreof the plurality of images and metadata associated with one or more ofthe plurality of images. In some implementations, the method furthercomprises automatically unarchiving at least one of the one or morearchived functional images in response to one or more unarchivingcriteria being met.

In some implementations, a system includes a memory and at least oneprocessor configured to access the memory and configured to performoperations. The operations include accessing an image and determiningone or more functional labels corresponding to the image and one or moreconfidence scores corresponding to the one or more functional labels.The operations include determining a functional image category for theimage and a functional image score for the image based on the one ormore functional labels having a corresponding confidence score thatmeets a respective threshold for the one or more functional labels. Theoperations include, in response to determining that the functional imagescore meets a functional image score threshold, providing a functionalimage signal that indicates the image is a functional image. Theoperations include generating an identifier associated with the imagebased on the functional image signal, where the identifier includes thefunctional image category, and archiving the image in response to thefunctional image category being an archiving category and the functionalimage score meeting an archiving threshold.

In various implementations of the system, the at least one processor isfurther configured to perform operations comprising determining andcausing output of a recommendation to archive the functional image basedon the functional image score, where the operation of archiving theimage is in response to user input received in response to the providingthe recommendation. In some implementations, the operation ofdetermining one or more functional labels corresponding to the image andthe one or more confidence scores corresponding to the one or morefunctional labels includes determining whether the one or morefunctional labels match predetermined labels associated with afunctional image designation.

In some implementations, a non-transitory computer readable medium hasstored thereon software instructions that, when executed by a processor,cause the processor to performing operations including accessing aplurality of images, determining one or more functional labelscorresponding to each of the plurality of images and one or moreconfidence scores corresponding to the one or more functional labels,and determining a functional image score for each of the plurality ofimages based on the one or more functional labels having a correspondingconfidence score that meets a respective threshold for the one or morefunctional labels. The operations include, in response to determiningthat the functional image score meets a functional image scorethreshold, providing a functional image signal that indicates that oneor more of the plurality of images that meet the functional image scorethreshold are one or more functional images. The operations includedetermining whether one or more images of the plurality of images areincluded in one or more functional image categories, and, in response todetermining that the one or more images of the plurality of images areincluded in one or more functional image categories, determining thatthe one or more functional images are to be archived. The operationsinclude causing the one or more functional images to be archived byassociating an archive attribute with the one or more functional imagessuch that the one or more functional images having the archive attributeare excluded from display in views of the plurality of images.

In various implementations of the computer readable medium the operationof determining one or more functional labels and one or more confidencescores includes programmatically analyzing one or more pixels of each ofthe plurality of images to detect content of the respective images,identifying one or more labels corresponding to each of the plurality ofimages based on the content of the image, determining whether one ormore functional labels are present within the one or more labels, and,if one or more functional labels are present, selecting the one or morefunctional labels and the one or more corresponding confidence scores toassociate with one or more respective images of the plurality of images.In some implementations, further operations include determining arecommendation to archive the functional images based on respectivefunctional image scores, and causing output of the recommendation by adevice, where determining that the one or more functional images are tobe archived is based on user input received at the device in response tothe providing the recommendation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of example systems and a network environmentwhich may be used for one or more implementations described herein;

FIG. 2 is a flow diagram illustrating an example method of functionalimage archiving, according to some implementations;

FIG. 3 is a flow diagram illustrating an example method of functionalimage archiving, according to some implementations;

FIG. 4 is a block diagram of an example computing device which may beused for one or more implementations described herein; and

FIG. 5 is a diagrammatic illustration of an example of user interface toprovide a suggestion to archive one or more images, according to someimplementations.

DETAILED DESCRIPTION

Some implementations include methods and systems to determine whichimages among an incoming stream of images are functional in nature andto archive those functional images and/or suggest archiving thosefunctional images. Images can include single images (e.g., photos),animated images, videos, graphics, etc. For example, a user may capturean image of a receipt from a store for warranty or other purposes. Thesystem can detect that this image is likely a functional image of areceipt and generate a functional image signal indicating that the imageis a functional image. Some additional examples of functional imagesinclude images of checks, medical prescriptions, advertisements, printedmaterial, newspapers, product labels, restaurant menus, handwrittennotes, instruction manuals, business cards, driver's licenses, passports(and other person identifications), screenshots, whiteboard orblackboard, memes, home projects (e.g., images of building or projectmaterials), etc. Functional images can include images taken by a userfor functional purposes, such as reminders, to store information, etc.

Some examples of non-functional images can include images that depicthuman faces or persons, images that have photographic characteristics,images with artistic composition or other creative attributes, images ofanimals, images of geographic scenery, images captured with certainsettings (e.g., images with depth of focus), etc. Based on thefunctional image signal, the system can perform an action such as make arecommendation to archive the image, automatically archive the image,etc.

An image as referred to herein can include a digital image having pixelswith one or more pixel values (e.g., color values, brightness values,etc.). An image can be a static image (e.g., a single frame with nomotion), a dynamic image (e.g., animated image, animated GIFs,cinemagraphs where a portion of the image includes motion while otherportions are static, etc.), a video (e.g., a sequence of a plurality ofimages or image frames that may include audio), etc. For example, astatic image may depict one or more objects, while an animated image maydepict one or more objects that change within the image (e.g., a livephoto that captures a face with eyes transitioning between closed andopen, face with the mouth moving from a non-smiling to a smilingposition, etc.). A video may include a plurality of frames that depictone or more persons.

While the remainder of this document refers to an image as a staticimage, it may be understood that the techniques described herein areapplicable for dynamic images, video, etc. For example, implementationsdescribed herein can be used with still images (e.g., a photograph, anemoji, or other image), videos, or dynamic images). Functional imagearchiving suggestions may be provided for any type of image and mayinclude a group of images of same and/or different types. Text, asreferred to herein, can include alphanumeric characters, emojis,symbols, or other characters.

Archiving an image includes designating the image as archived, e.g., byassociating the image with data (e.g., metadata or other associateddata) that indicates the image has been archived, e.g., data such as anarchive attribute. In various implementations, archived images can beprocessed, displayed, stored, etc. differently than non-archived imagesbased on the archived status. In one or more implementations describedherein, archived images are hidden from display in displayed views ofnon-archived images (unless user input is received that instructsarchived images to be displayed in such a view). For example, archivedimages can be excluded from being displayed in a main gallery or primarydisplay of images.

In some implementations, images are analyzed to determine a functionalimage score. The analyzing can include performing image content analysisto determine content of the image, determining one or more labels to theimage based on the content, and identifying functional labels within theone or more labels applied to the image. The labels can include anassociated confidence score that can be thresholded to identify labelswith highest confidence. Within the labels, functional labels can beidentified, e.g., by looking up a list of predetermined labelsassociated with functional images and/or by using a machine-learningmodel trained to identify labels associated with functional images.Analyzing the images for functional image score can also includeanalyzing metadata in some implementations. Some implementations caninclude determining a functional image score for each image in aplurality of images, sorting the images according to the functionalimage score, and selecting a number of images (e.g., the top 5, the top10, or a top percentage) for archival recommendation.

The functional image signal may be generalized to an indication of animage that is functional in nature. For example, in addition to beinguseful for archiving suggestions and/or automatically archiving, asignal that indicates that an image is likely a functional image can beused for excluding or filtering functional images from automaticallygenerated creations (e.g., photo books, image animations, collages,stories, etc.).

The recommendation to archive an image (or images) can be surfaced (ordisplayed) as a suggestion card (e.g., FIG. 5) in a graphical userinterface (e.g., as part of an assistant application, when a photosapplication is opened, at a given time interval, etc.) or displayed as achip (selectable user interface element), e.g., on or near a singleimage being viewed in a user interface such as a user interface of animage display or editing application or a camera application. In someimplementations, the functional image analysis may be performed a giventime period after the image was acquired (e.g., one week later, etc.).

In some implementations, functional image archiving suggestions can bepersonalized to user behavior (e.g., using a machine-learning model asdescribed herein) to suggest archiving more of the types of images thatthe user archives and less of the types of utility photos that the userdoes not archive, for example.

When making functional image archiving suggestions to a user, it may behelpful for an image system to make functional image archivingsuggestions that are in line with a user's previous functional imagearchiving activity. To make functional image archiving suggestions, aprobabilistic model (or other model as described below in conjunctionwith FIG. 4) can be used to make an inference (or prediction) about howlikely an image is to be a functional image and how likely a user is toarchive an image or group of images. Accordingly, it may be helpful tomake an inference regarding a probability that an image is a functionalimage.

The inference based on the probabilistic model can include predicting awhether an image is a functional image, determining a confidence scorefor the prediction, and making functional image archiving suggestions inaccordance with functional image and confidence score as inferred fromthe probabilistic model. The probabilistic model can be trained withdata including previous functional image archiving activity of one ormore users. Some implementations can include generating a functionalimage archiving suggestion for one or more images having a functionalimage score based on objects in the image or data associated withfunctional images. The functional image score may be based on aninference from a probabilistic model that is trained using data forwhich respective users have provided permission for use in training theprobabilistic model. Such data may include user image data and imageactivity data (e.g., archiving data). Each image can be annotated withone or more labels and a confidence score for a respective label.

An implementation of the system or method described herein can beintegrated with an image management system. For example, a probabilisticmodel for functional image archiving as described herein can be added toa configuration of an image management system or framework (e.g., at aserver system, a cloud based computing system, or within a user devicesuch as a mobile device, laptop or desktop computer).

The systems and methods provided herein may overcome one or moredeficiencies of some conventional image management systems and methods.For example, image management includes enabling users to capture, store,and/or share images (e.g., digital images represented as data stored ina nontransitory computer readable medium) with other users. With theeasy availability of digital image capture devices, such as digitalcameras, phones with built-in cameras, wearable devices with cameras,head-mounted devices, tablets, personal computers, etc., users maycapture a large number (e.g., a hundred, a thousand, etc.) of images.Conventional image management systems may enable users to manuallyarchive images. Conventional image management systems may permitarchiving images upon user commands to archive the images. For example,a user may have functional images interspersed with images from asignificant event, e.g., a wedding, a graduation, etc. with other users,e.g., friends, family members, colleagues, etc. Manually archiving thefunctional images in some conventional system may require the user tomanually look through numerous images and manually archive eachfunctional image. In this manner, conventional image management systemsmay not provide automatic functional image archiving and/or functionalimage archiving suggestions, where sharing such functional imagearchiving and/or suggestions for functional image archiving may beuseful to the user.

The example systems and methods described herein may overcome one ormore of the deficiencies of conventional image management systems toprovide users functional image archiving. A technical problem of someconventional image management systems may be that such systems do notsuggest functional image archiving of images based on functional imagelabels and corresponding confidence scores. In conventional imagemanagements systems, views of images, e.g., a main view, may displayimages in various layouts, e.g., as a grid. In these systems, functionalimages may be displayed alongside images that are not functional, e.g.,photographs or videos that depict people, scenery, monuments, or otherobjects of photographic interest. Display of functional images in suchviews may provide an undesirable user experience. For example, a viewthat displays a user's photographs from a vacation may include imagesthat the user captured of restaurant receipts, tickets, boarding passes,and other functional images that are not of interest when the user isreviewing the vacation, e.g., at a later time. Conventional systemsinclude such images in the view, e.g., in a chronological view andrequire the user to manually filter such images.

The disclosed subject matter relates to particular techniques to archiveor to generate suggestions to archive a group of functional images,e.g., as a single image, a group of images, or an image composition suchas an image album, video, collage, etc. The suggestions are based oninstantiating a process on a computer to determine a functional imagescore representing whether an image is a functional image, where thefunctional image score is determined based on the image and/or dataassociated with the image. The process on the computer can determine oneor more images to be recommended for archiving based on objects in theimages and prior images that were archived.

The systems and methods presented herein automatically providesuggestions to archive functional images that are more likely to beaccepted by users and that likely contain functional images. Particularimplementations may realize one or more of the following advantages. Anadvantage of generating suggestions for functional image archiving basedon methods and system described herein is that images likely suitablefor archiving are presented to the user automatically, without the userhaving to browse image collections, select images, designate archivalstatus to images, etc., thus saving device resources such as displayprocessing and other processing, memory, etc. Another advantage is thatthe suggestions are based on a functional image score and confidence.Another advantage is that, if an image is not found to be functional,image archiving suggestions may not be provided, thereby reducing oreliminating generation of archiving suggestions that a user may notapprove, which can result in fewer processing operations and thus reducelatency in the overall system. A further advantage of someimplementations is that the suggestion can be based on a user's previousarchiving activity, which can permit the image archiving suggestions tobe tailored to different users. Yet another advantage is that themethods and systems described herein can dynamically learn newthresholds (e.g., for confidence scores, etc.) and provide suggestionsfor images that match the new thresholds.

Archiving of functional images is described herein to help illustratethe disclosed subject matter. Functional images can include images ofdocuments (e.g., receipts, documents, tickets, etc.) physical objects(e.g., product labels, whiteboards, parking location, screenshots,license plates, etc.), identification documents (e.g., passports,driver's licenses, identification cards, business cards, etc.), and/ortransactional cards (e.g., credit cards, loyalty program cards, librarycards, discount cards, etc.). It will be appreciated that the disclosedfunctional image archiving can be applied to images other thanfunctional images, such as poor quality images (e.g., pocket shots,blurry images, poorly exposed images, etc.), near duplicates (e.g., thelowest quality near duplicate images), and gifted images (e.g., memes,etc., which can include images received via a social network, messagingservice, or the like).

In some implementations, detected functional images can be marked byadding an identifier that is indexed by functional image category and/orconfidence score. Images can be marked as archive suggestions during anupload process. Archiving suggestions can be fetched by category andminimum confidence. In some implementations, each image may havesuggestions of multiple types, but only one category (e.g., functionalor gifted) per type (e.g., receipt category in the functional type).Other implementations can provide multiple categories per type.Suggestions can be separated into categories to help facilitateautomated archiving for user-determined (or machine-learning modeldetermined) reliable categories. Suggestions can also be marked asaccepted/dismissed when users act on the suggestions. In someimplementations, a single suggested archive card may be displayed orsurfaced within a graphical user interface. The suggested archive cardcontents and position (e.g., in an assistant view) can be updated assuggestions are added (uploaded) or removed (deleted). Content of asuggested archive card can be populated by a notification renderer.Latest suggestions to archive can be fetched upon request to see thecard to account for any manual archiving or suggestion deletion andinsertion. Dismissed suggested archive cards can imply rejecting one ormore archive suggestions contained in the card and the rejection can beused as training data to train and/or tune a machine-learning model asdiscussed herein.

Some implementations can include generating a functional image signalthat represents a prediction of whether a particular image suggested forarchiving will be archived by a user. The functional image signal caninclude a single-source prediction or a multiple-source prediction(e.g., determining a functional image signal based on multiple images).A single-source functional image prediction signal is based on a singleimage. Single-source predictions can be used for functional imagesand/or different categories of images, etc. Multiple-source functionalimage signal prediction can be used, for example, for worse duplicatesarchiving suggestions (e.g., recommending lower quality duplicate ornear-duplicate images for archiving, such as images captured in a burstor near in time at the same location).

FIG. 1 illustrates a block diagram of an example network environment100, which may be used in some implementations described herein. In someimplementations, network environment 100 includes one or more serversystems, e.g., server system 102 in the example of FIG. 1. Server system102 can communicate with a network 130, for example. Server system 102can include a server device 104 and a database 106 or other data storeor data storage device. Network environment 100 also can include one ormore client devices, e.g., client devices 120, 122, 124, and 126, whichmay communicate with each other and/or with server system 102 vianetwork 130. Network 130 can be any type of communication network,including one or more of the Internet, local area networks (LAN),wireless networks, switch or hub connections, etc. In someimplementations, network 130 can include peer-to-peer communication 132between devices, e.g., using peer-to-peer wireless protocols.

For ease of illustration, FIG. 1 shows one block for server system 102,server device 104, and database 106, and shows four blocks for clientdevices 120, 122, 124, and 126. Some blocks (e.g., 102, 104, and 106)may represent multiple systems, server devices, and network databases,and the blocks can be provided in different configurations than shown.For example, server system 102 can represent multiple server systemsthat can communicate with other server systems via the network 130. Insome examples, database 106 and/or other storage devices can be providedin server system block(s) that are separate from server device 104 andcan communicate with server device 104 and other server systems vianetwork 130. Also, there may be any number of client devices. Eachclient device can be any type of electronic device, e.g., desktopcomputer, laptop computer, portable or mobile device, camera, cellphone, smart phone, tablet computer, television, TV set top box orentertainment device, wearable devices (e.g., display glasses orgoggles, head-mounted display (HMD), wristwatch, headset, armband,jewelry, etc.), virtual reality (VR) and/or augmented reality (AR)enabled devices, personal digital assistant (PDA), media player, gamedevice, etc. Some client devices may also have a local database similarto database 106 or other storage. In other implementations, networkenvironment 100 may not have all of the components shown and/or may haveother elements including other types of elements instead of, or inaddition to, those described herein.

In various implementations, end-users U1, U2, U3, and U4 may communicatewith server system 102 and/or each other using respective client devices120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 mayinteract with each other via applications running on respective clientdevices and/or server system 102, and/or via a network service, e.g., animage sharing service, a messaging service, a social network service orother type of network service, implemented on server system 102. Forexample, respective client devices 120, 122, 124, and 126 maycommunicate data to and from one or more server systems (e.g., serversystem 102). In some implementations, the server system 102 may provideappropriate data to the client devices such that each client device canreceive communicated content or shared content uploaded to the serversystem 102 and/or network service. In some examples, the users caninteract via audio or video conferencing, audio, video, or text chat, orother communication modes or applications. In some examples, the networkservice can include any system allowing users to perform a variety ofcommunications, form links and associations, upload and post sharedcontent such as images, image compositions (e.g., albums that includeone or more images, image collages, videos, etc.), audio data, and othertypes of content, receive various forms of data, and/or performsocially-related functions. For example, the network service can allow auser to send messages to particular or multiple other users, form sociallinks in the form of associations to other users within the networkservice, group other users in user lists, friends lists, or other usergroups, post or send content including text, images, image compositions,audio sequences or recordings, or other types of content for access bydesignated sets of users of the network service, participate in livevideo, audio, and/or text videoconferences or chat with other users ofthe service, etc. In some implementations, a “user” can include one ormore programs or virtual entities, as well as persons that interfacewith the system or network.

A user interface can enable display of images, image compositions, data,and other content as well as communications, privacy settings,notifications, and other data on client devices 120, 122, 124, and 126(or alternatively on server system 102). Such an interface can bedisplayed using software on the client device, software on the serverdevice, and/or a combination of client software and server softwareexecuting on server device 104, e.g., application software or clientsoftware in communication with server system 102. The user interface canbe displayed by a display device of a client device or server device,e.g., a display screen, projector, etc. In some implementations,application programs running on a server system can communicate with aclient device to receive user input at the client device and to outputdata such as visual data, audio data, etc. at the client device.

In some implementations, server system 102 and/or one or more clientdevices 120-126 can provide a functional image archiving program. Thefunctional image archiving program may allow a system (e.g., clientdevice or server device) to provide suggestions for archiving imagesthat are likely to be functional images, some examples of which aredescribed herein. The functional image archiving program can provideassociated user interface(s) that are displayed on a display deviceassociated with the server system or client device. The user interfacemay provide various options to a user to select archive images, etc.Other applications can also be used with one or more features describedherein, such as image management applications, browsers, emailapplications, communication applications, etc.

Various implementations of features described herein can use any type ofsystem and/or service. For example, social networking services, imagecollection and sharing services, assisted messaging services or othernetworked services (e.g., connected to the Internet) can include one ormore described features accessed by client and server devices. Any typeof electronic device can make use of features described herein. Someimplementations can provide one or more features described herein onclient or server devices disconnected from or intermittently connectedto computer networks. In some examples, a client device including orconnected to a display device can examine and display images stored onstorage devices local to the client device (e.g., not connected via acommunication network) and can provide features and results as describedherein that are viewable to a user.

FIG. 2 is a flow diagram illustrating an example method 200 to generatea functional image signal and optionally perform one or more actionsbased on the functional image signal, according to some implementations.

In some implementations, method 200 can be implemented, for example, ona server system 102 as shown in FIG. 1. In other implementations, someor all of the method 200 can be implemented on one or more clientdevices (e.g., client devices 120, 122, 124, or 126 as shown in FIG. 1),one or more server devices, and/or on both server device(s) and clientdevice(s). In described examples, the implementing system includes oneor more digital hardware processors or processing circuitry(“processors”), and one or more storage devices (e.g., a database 106 orother storage). In some implementations, different components of one ormore servers and/or clients can perform different blocks or other partsof the method 200.

Some implementations can initiate method 200 based on user input. A usermay, for example, have selected the initiation of the method 200 from adisplayed user interface. In some implementations, method 200 orportions thereof can be performed with guidance by the user via userinput.

In some implementations, the method 200, or portions of the method, canbe initiated automatically by a device. For example, the method (orportions thereof) can be periodically performed or performed based onthe occurrence of one or more particular events or conditions. Forexample, such events or conditions can include a particular applicationbeing opened by a user, an elapsed time since obtaining one or moreimages that have been captured by, uploaded to, or otherwise accessibleby a device (e.g. a user device), a predetermined time period havingexpired since the last performance of method 200, and/or one or moreother events or conditions occurring which can be specified in settingsof a device implementing method 200. In some implementations, suchconditions can be previously specified by a user in stored custompreferences of the user (accessible by a device or method with userconsent). In some examples, a device (server or client) can perform themethod 200 with access to a large collection of accessible images, e.g.,a user's collection of images (if user consent is received). In anotherexample, a camera, cell phone, tablet computer, wearable device, orother client device can capture one or more images and can perform themethod 200. In addition, or alternatively, a client device can send oneor more captured images to a server over a network, and the server canprocess the images using method 200.

In block 202, it is checked whether user consent (e.g., user permission)has been obtained to use user data in the implementation of method 200.For example, user data can include user preferences, user images in animage collection (e.g., images captured by a user, uploaded by a user,or otherwise associated with a user), information about a user's socialnetwork and/or contacts, user characteristics (identity, name, age,gender, profession, etc.), social and other types of actions andactivities, calendar and appointments, content, ratings, and opinionscreated or submitted by a user, a user's geographical location,historical user data, etc. One or more blocks of the methods describedherein may use such user data in some implementations. Block 202 may beperformed as part of a sharing suggestion framework and/or to verifyconsent provided at the functional image archiving suggestion frameworklevel such that blocks 204 and on will only be invoked if user consentfor performing the functional image archiving suggestions was obtainedat the functional image archiving suggestion framework level. If userconsent has been obtained from the relevant users for which user datamay be used in the method 200, then in block 204, it is determined thatthe blocks of the methods herein can be implemented with possible use ofuser data as described for those blocks, and the method continues toblock 206. If user consent has not been obtained, it is determined inblock 206 that blocks are to be implemented without use of user data,and the method continues to block 206. In some implementations, if userconsent has not been obtained, the remainder of method 200 is notperformed, and/or particular blocks needing the user data are notperformed.

In block 208, image data is obtained for one or more images. Image datacan include image pixel data (e.g., image content) and/or otherassociated image data (e.g., metadata) including information about when(e.g., time and date stamp), how (e.g., imaging device type andsettings, etc.) and where (e.g., location data) an image was acquired.The image data can also optionally include one or more labelsidentifying respective objects or attributes within the image that havepreviously been associated with the image. The labels may include acorresponding confidence value representing a statistical confidence (orother type of confidence value) of the accuracy of the label withrespect to the image. The method continues to 210.

In block 210, the image data is programmatically analyzed. For example,at least a portion of the image data may be programmatically analyzed byan image content analysis (ICA) system that identifies features (e.g.,objects, attributes, landscape features, and/or other features) depictedwithin the image data (e.g., depicted in image pixels for visualdisplay). The ICA system can be part of the functional image archivingsystem, can be part of the image management system that includes thefunctional image archiving system, or can be a separate system.Programmatically analyzing the image may include analyzing the imageusing a trained machine learning model that is trained to identifyfeatures within the image, e.g., using image pixel data and/or imagemetadata. The method continues to block 212.

In block 212, one or more labels corresponding to the image aredetermined. For example, the image content analysis system may providean indication of one or more labels associated with the image based onanalysis of the image data (e.g., based on analysis of image contentand/or metadata). For example, the labels can be determined based on thecontent of the image, e.g., one or more features identified in the imagedata in block 210. In some implementations, programmatically analyzingimage data and determining labels for one or more images can beperformed on the client side as a cloud service. In someimplementations, a low resource image content analysis model can bedeployed on a client device (e.g., a mobile device). The low resourceimage content analysis model may use a simplified approach toprogrammatically analyzing an image and determining labels for theimage. For example, the low resource image content analysis model mayidentify that a quantity of text in a photo meets a threshold and usethat signal in isolation to determine that the image is a functionalimage. In another example, the functional image archiving system maydetermine the one or more labels based on a result from the programmaticanalysis performed by the image content analysis system. The one or morelabels can also include one or more corresponding confidence scores. Insome implementations, a trained machine learning model may be used todetermine one or more labels for the image. In some implementations, thelabels may correspond to semantic concepts, e.g., semantic conceptsbased on the features identified in the image. In some implementations,the semantic concepts may be organized in a hierarchy, e.g.,“sports->baseball->baseball bat,” etc. The method continues to block214.

In block 214, a functional image score for the image is determined. Thefunctional image score can indicate the suitability of the image forarchiving, e.g., the score can represent a predicted likelihood of auser designating the image for archiving.

In some implementations, a machine-learning model can determine afunctional image score for an image based on labels that the model hasdetermined are related to functional images and on threshold valuesdetermined by the model for each of the labels related to functionalimages. For example, the model can be (previously) trained based onimages having functional content. In some implementations, other modelsor techniques can be used to determine the functional image score, e.g.,statistical methods, graph-based models, etc.

In some implementations, the label confidence scores of the labels forthe image can be used in the determination of the functional imagescore. In some implementations, as part of functional image scoredetermination, the label confidence scores may be mapped from a firstspace (e.g., the image content analysis space) to a second space (e.g.,a functional image archiving score space). The mapping can be performedto adjust for the potential difference between a likelihood in the imagecontent analysis space that an object/feature is identified accurately,and a likelihood in the functional imaging archive space that thelabeled object/feature represents a functional image that will likely bearchived. Mapping of a score from image content analysis space tofunctional image archiving space can improve accuracy for archivalpurposes. For example, a probability of an object being in an image maybe different from a probability of archiving an image with that object(e.g., a 0.2 image content analysis space may correspond to 0.8 in afunctional image archive space of a suggestion model). A calibration mayneed to be performed when an image content analysis system is updated toupdate the mapping from the image content analysis space to thefunctional image archiving space. The calibration can be confined toupdating the image content analysis system and performing thecalibration. By using the calibration technique, the thresholds infunctional image archiving score space may be unaffected by an imagecontent analysis system update. The calibration may be performed by amachine-learning model.

In some implementations, one or more functional labels (andcorresponding functional image archive space confidence scores) may beidentified from the one or more labels associated with the image. Insome examples, a functional label is a label that associated withfunctional images. One or more of the labels can be identified asfunctional labels, e.g., by comparing the labels to a list ofpredetermined labels associated with a functional image designation andselecting the labels that match, e.g., have an exact match with or asemantic match with one or more of the predetermined labels. In someimplementations, functional labels can be identified among the labels byusing a machine-learning model that has been trained to identify labelsassociated with functional images.

In some implementations, the corresponding confidence scores of thelabels can be thresholded to identify labels with higher confidencescores (e.g., scores above a threshold) and/or highest confidence scoresamong the labels (e.g., the scores indicating the greater or greatestconfidence in the accuracy of the associated labels). These identifiedlabels can be used to determine the functional image score. In someimplementations, the corresponding confidence scores of functionallabels (identified as described above) can be thresholded in this mannerto determine the functional labels to be used in determining thefunctional image score. For example, the thresholded confidence scorescan be functional image archive space confidence scores as describedabove.

In some implementations, the functional image score can be based on oneor more of the functional labels and corresponding functional imagearchive space confidence scores. In one example, the functional imagescore can be based, at least in part, on the confidence scoresassociated with functional labels, e.g., a higher functional image scoreis determined from higher confidence scores. In some implementations,the functional image score can be based, at least in part, on the amountof (thresholded) functional labels identified for the image, e.g., thegreater the number of such functional labels, the more that thefunctional image score indicates that the image is suitable forarchiving (e.g., a greater functional image score can indicate a higherlikelihood of archiving). In some implementations, particular functionallabels can be designated to have greater weight and provide a greatercontribution to the functional image score than other particularfunctional labels (e.g., a greater value added to a functional imagescore total). For example, a functional label of “text” can cause agreater increase in functional image score than a functional label of“paper sheet.”

In some implementations, a functional image score can be based at leastin part on locations of detected content features within an image,and/or other visual contexts of detected features. For example, if aface is detected to one side of an image, is surrounded by a squareframe, and is located next to text, then the functional image score canbe increased, e.g., since the face may be part of a functional image ofa passport, driver's license, or other personal identification thattypically have such a layout.

Determining the functional image score can also be based on metadata ofthe image, in some implementations. For example, metadata can be used toadjust a functional image score. In some examples, geographical locationof capture of the image can influence the functional image score, basedon the type of location, e.g., home, business or store, etc. (e.g., abusiness or store location can adjust the functional image score toindicate greater likelihood that the image is a functional image). Insome implementations, other types of image metadata can be used todetermine the functional image score, e.g., EXIF data (describingsettings or characteristics of a camera capturing the image), timestamp,etc.

When a plurality of images are being analyzed, some implementations caninclude determining a functional image score for each image of theplurality of images, sorting the images according to the functionalimage score, and selecting a number of images (e.g., the top 5, the top10, or a top percentage) for archival recommendation.

In some implementations, a blacklist and/or a whitelist can be employed.The blacklist or whitelist may include particular content features that,if detected as present in an image, automatically disqualify the imagefrom being a functional image (e.g., if using a blacklist) orautomatically qualify the image to be functional image (e.g., if using awhitelist). For example, a blacklist can include content features suchas faces, person, child, etc.

In some implementations, one or more functional image categories canalso be determined for the image, e.g., in block 214 (or other block ofthe method 200 or method 300 described below). In some examples, theimage can be determined to be included in the one or more functionalimage categories based on the labels determined for the image asdescribed above. In various implementations, a determined functionalimage category can be a broad category (e.g., “functional image”) or amore specific category (e.g., “receipt,” “business card,” etc.). In someimplementations, each functional image category can be associated withone or more category labels, such that if one or more labels of theimage match (e.g., exactly match or semantically match) one or morecorresponding category labels, the image can be designated to beincluded in that category. In some examples, if a threshold number oflabels of an image match category labels of a category, then the imagecan be designated to be included in that category. For example, a“receipt” category can be associated with category labels such as“text,” “numbers,” “prices,” “sheet of paper,” etc., and the image canbe determined to be included in the “receipt” category based on havingone or more labels that match these category labels. In someimplementations, a machine-learning model can be used to determine afunctional image category for an image based on labels that the modelhas determined are related to that functional image category. Forexample, the model can be (previously) trained based on images havingfeatures (labels) that are included in designated functional categories.

In some implementations, the image can be determined be included inmultiple functional image categories, e.g., a “receipt” category and a“document” category. In some implementations, a category confidencescore is determined for each category determined for the image, wherethe category confidence score indicates a likelihood or confidence thatthe image belongs to that category (e.g., based on the confidence scoresassociated with the predefined labels that are associated with thatcategory, based on the number of labels that match the predefined labelsassociated with that category, and/or based on output of a machinelearning model that has been trained with multiple images of thatcategory). In some implementations, a single category is determined foran image, which can be selected, for example, as the category having thehighest category confidence score.

In some implementations, a functional image identifier can be generatedand associated with one or more of the images, e.g., can be associatedwith images that are included in all or particular functional imagecategories, or associated with all of the images. For example, thefunctional image identifier can be metadata or other associated data.The functional image identifier specifies one or more functional imagecategories for each associated image and a category confidence scoreassociated with each of the functional image categories. In someimplementations, the functional image identifier can specify image type,which can be such types as “functional”, “non-functional”, or othertypes, etc. The functional image identifier can be generated and/orassociated with images at any of various stages, e.g., prior to method200, during one or more blocks of method 200, in method 300 (describedbelow), etc. The method continues to 216.

In block 216, a functional image signal is provided if the functionalimage score determined in block 214 meets a functional image scorethreshold. The functional image signal indicates that the image is afunctional image. For example, the functional image signal can include asoftware signal, e.g., designation or attribute for the image thatindicates that the image meets the functional image score threshold. Insome implementations, a functional image signal can be associated withmultiple images, indicating that each of those images is a functionalimage.

In some implementations, the functional image signal can be provided toanother process that can optionally act on the functional image signal.For example, one or more actions can be performed based on thefunctional image signal. For example, as described below in method 300,actions can be performed on images associated with a functional imagesignal, including, for example, auto-archiving, archiving based on userinput, or deleting the images. In some implementations, the functionalimage scores of the associated images can be provided with thefunctional image signal. In some implementations, information indicatingone or more determined functional image categories, and categoryconfidence scores for those categories, can also be provided inassociation with the functional image signal for each of the associatedimages (e.g., if the image meets the functional image score threshold),such as the functional image identifier. In some implementations, someor all such information can be associated and accessed with the image,e.g., as metadata.

In some implementations, in addition to the use of a low resource imagecontent analysis model as described above, a low resource functionalimage score model can be used that can include a simplified model forgenerating the functional image signal.

In FIG. 2, various blocks (e.g., blocks 202-218) are illustrated asbeing performed sequentially. It will be appreciated however that theseblocks may be re-arranged as convenient to suit particular embodimentsand that these blocks or portions thereof may be performed concurrentlyin some embodiments. It will also be appreciated that in some examplesvarious blocks may be eliminated, divided into additional blocks, and/orcombined with other blocks.

Blocks 214 and 216 of FIG. 2 describe optionally utilizing aprobabilistic model (or other type of model as described above or, e.g.,in conjunction with FIG. 4) to determine image labels and to generate afunctional image score, respectively. In some implementations, whenusers permit the use of their data for sharing, the probabilistic modelcan be based on shared data from such users. For example, with userconsent, different probabilities may be determined for each user accountand be used to determine the probability that a given image may bearchived. The probabilities may be aggregated (e.g., across a pluralityof users that provide consent to such use of their sharing data) toprovide a statistical distribution for archiving probabilities. Suchstatistical distribution can be used in the probabilistic model. Ingenerating the statistical distribution, user accounts of users who donot consent to such use of sharing data are excluded. Further, metadataregarding whether an image is shared and whether the image includes atop cluster may be used, and no use is made of image content.

FIG. 3 is a flow diagram illustrating an example method 300 forfunctional image archiving, according to some implementations.

In some implementations, method 300 can be implemented, for example, ona server system 102 as shown in FIG. 1. In some implementations, some orall of the method 300 can be implemented on one or more client devices(e.g., client devices 120, 122, 124, or 126 as shown in FIG. 1), one ormore server devices, and/or on both server device(s) and clientdevice(s). In described examples, the implementing system includes oneor more digital hardware processors or processing circuitry(“processors”), and one or more storage devices (e.g., a database 106 orother storage). In some implementations, different components of one ormore servers and/or clients can perform different blocks or other partsof the method 300.

Some implementations can initiate method 300 based on user input. A usermay, for example, have selected the initiation of the method 300 from adisplayed user interface. In some implementations, method 300 orportions thereof can be performed with guidance by the user via userinput.

In some implementations, the method 300, or portions of the method, canbe initiated automatically by a device. For example, the method (orportions thereof) can be periodically performed or performed based onthe occurrence of one or more particular events or conditions. Forexample, such events or conditions can include a particular applicationbeing opened by a user, obtaining one or more images that have beennewly captured by, uploaded to, or otherwise accessible by a device(e.g., a user device), a predetermined time period having elapsed sincesuch images were obtained, a predetermined time period having expiredsince the last performance of method 300, and/or one or more otherevents or conditions occurring which can be specified in settings of adevice implementing method 300. In some implementations, such conditionscan be previously specified by a user in stored custom preferences ofthe user (accessible by a device or method with user consent). In someexamples, a device (server or client) can perform the method 300 withaccess to a large collection of accessible images, e.g., a user'scollection of images (if user consent is received). In another example,a camera, cell phone, tablet computer, wearable device, or other clientdevice can capture one or more images and can perform the method 300. Inaddition, or alternatively, a client device can send one or morecaptured images to a server over a network, and the server can processthe images using method 300.

In block 302, it is checked whether user consent (e.g., user permission)has been obtained to use user data in the implementation of method 300.For example, user data can include user preferences, user images in animage collection (e.g., images captured by a user, uploaded by a user,or otherwise associated with a user), information about a user's socialnetwork and/or contacts, user characteristics (identity, name, age,gender, profession, etc.), social and other types of actions andactivities, calendar and appointments, content, ratings, and opinionscreated or submitted by a user, a user's geographical location,historical user data, etc. One or more blocks of the methods describedherein may use such user data in some implementations. Block 302 may beperformed as part of a functional image archiving and/or functionalimage archiving suggestion framework and/or to verify consent providedat the sharing suggestion framework level such that blocks 304 and onwill only be invoked if user consent for performing the sharingsuggestions was obtained at the functional image archiving suggestionframework level. If user consent has been obtained from the relevantusers for which user data may be used in the method 300, then in block304, it is determined that the blocks of the methods herein can beimplemented with possible use of user data as described for thoseblocks, and the method continues to block 306. If user consent has notbeen obtained, it is determined in block 306 that blocks are to beimplemented without use of user data, and the method continues to block306. In some implementations, if user consent has not been obtained, theremainder of method 300 is not performed, and/or particular blocksneeding the user data are not performed.

In block 308, a functional image signal is obtained for one or moreimages being accessed. For example, the functional image signal can beobtained as a signal accompanying the one or more images when the one ormore images are received as input to an image management program. Inanother example, the functional image signal can be obtained in responseto a given amount of time elapsing since the one or more images werecaptured or acquired (e.g., 7 days, 30 days, etc.). In another example,the functional image signal can be obtained in response to a request forfunctional image analysis on the one or more images prior to a systemautomatically generating a media creation such as a slide show or story(e.g., a format that includes one or more images, and other content suchas text, emojis, clipart, backgrounds, etc., optionally arranged inpredetermined layouts, e.g., on a webpage). In some implementations,functional image scores for the one or more images are obtained inassociation with the functional image signal. In some implementations,information indicating one or more determined functional imagecategories, and category confidence scores for those categories, for theone or more images can also be obtained in association with thefunctional image signal, e.g., a functional image identifier, asdescribed above, can be obtained. The method continues to block 310.

In block 310, one or more of the images are selected for archiving basedon one or more archiving criteria that are based on the functional imagesignal and/or elapsed time. For example, the archiving criteria canspecify that selected images should be associated with a functionalimage signal and have a functional image score that meets an archivingscore threshold. In various implementations, the archiving scorethreshold can be the same as the functional image score thresholddescribed above for FIG. 2, or can be a different threshold, e.g., athreshold requiring greater confidence (e.g., a higher threshold score).

In further examples, the archiving criteria can include elapsed timecriteria specifying that a given time period (e.g., 30 days) has elapsedsince the selected images were captured, first accessed, or obtained bythe auto-archiving system.

In further examples, the archiving criteria can specify that selectedimages should be included in one or more archiving categories designatedfor archiving (e.g., “receipts,” etc.). In various implementations, thearchiving categories can be the same as or a subset of the functionalimage categories described above. In some implementations, a functionalimage identifier is associated with each of the selected images (e.g.,as described with respect to FIG. 2), where the functional imageidentifier specifies one or more functional image categories for eachassociated image and a category confidence score associated with each ofthe functional image categories (and/or can specify image type, whichcan be such types as “functional,” “non-functional,” or other types,etc.). Such a functional image identifier can be examined to determineif selected images meet the archiving criteria specifying one or morearchiving categories.

In further examples, each archiving category can be associated with acategory confidence score threshold for that category, e.g., a minimumconfidence score for that category. The archiving criteria can specifythat a selected image should have a category confidence score that meetsthe category confidence score threshold for the particular archivingcategory (or categories) in which the image is included.

In some implementations, an auto-archiving system can request thatimages meeting particular archiving criteria be archived withoutrequiring user input or intervention (e.g., without providingsuggestions of block 312). In some implementations, such automaticarchival may be enabled for one or more particular archiving categories,e.g., if the precision is acceptably high for those particular archivingcategories (e.g., based on prior categorizations and labels determinedfor images). Different categories may have different precisioncharacteristics.

In some implementations, a user can enable or disable the selection ofimages for archiving and/or can configure the archiving criteria,categories, and thresholds. The method continues to block 312.

In block 312, in some implementations, a recommendation or suggestion toarchive the selected images is presented. In some examples, a userinterface is displayed to present the suggestion, e.g., an archivesuggestion card in a graphical user interface is surfaced or displayedon a device that includes the suggested images to archive (e.g., as partof an assistant application, when an image display or editingapplication is opened, at a given time interval, etc.). In someexamples, the suggested images can be represented by thumbnail images,titles, filenames, etc. In some implementations, a suggestion chipelement (user interface graphical element) is displayed in a userinterface, e.g., on or near a single image being viewed in the userinterface in an image viewing/editing interface, camera application,etc. The chip can include text asking whether the user wishes to archivethat single image. In other examples, the suggestion ship can bedisplayed within a user interface element (e.g., a button, text field,etc.) and/or in a particular location within the user interface, e.g.,the perimeter or center of the user interface, etc., where thesuggestion chip element can display a representation of one or more ofthe suggested images (e.g., thumbnails), filenames or titles of thesuggested images, etc. to archive.

In some implementations, the suggested images are presented in aparticular order, e.g., an order based on their functional image scores,category confidence scores, or other scores or ranking. In someexamples, a particular number of the selected images (e.g., the top 5,the top 10, or a top percentage) can be presented in the user interface,and the other selected images can be hidden until presented in responseto user input instructing such presentation. In some implementations,the presented suggested images meet an elapsed time period criterionthat may be different than the elapsed time period criterion used toselect the images in block 310. In some implementations, the archivesuggestion can include a display of the one or more determinedfunctional image categories in which the selected images belong. In someexamples, a single category can be displayed, e.g., the category towhich the greatest number of selected images belong. In some of theseimplementations, selected images that are not included in this categoryare not displayed as suggested images (e.g., a different functionalimage category can be displayed after user input is received, showingremaining selected images as suggestions that are included in thatcategory). In some implementations, the recommendation can include areason that the suggestion card element is being displayed for a givenimage where the reason is based on functional labels for the given imageand/or the functional image categories of the given image. For example,the suggestion card can include text such as “clean up your photolibrary by archiving photos of business cards” as a reason forsuggesting archival.

In some implementations, the archive suggestions are presented as adefault. In some implementations, the archive suggestions are notpresented in block 312 and the selected images are designated forautomatic archival without user input if particular criteria are met. Insome examples, a first archiving threshold can be at a higherrequirement level (e.g., a higher value) than a second archivingthreshold. An image can be designated for automatic archival withoutproviding the archive suggestions (and without receiving user input) ifthe functional image score of the image (or category confidence scorefor the archiving category of the image) meets the first archivingthreshold. If the functional image score of the image (or categoryconfidence score for the archiving category of the image) meets thesecond archiving threshold, and does not meet the first archivingthreshold, the selected images are presented as archiving suggestionsand user input is requested to accept, reject, and/or delay thesuggestions. The method continues to block 314.

In block 314, it is determined whether the selected images are to bearchived. In some implementations, the selected images are automaticallyarchived, without user input, if the selected images meet the archivingcriteria. In implementations which provide archive suggestions toarchive the selected images (e.g., as in block 312), the decision toarchive can be based on user input received from a user interface thataccepts, rejects, and/or delays the archiving suggestion (e.g., wheredelay input adds a time period before the selected images are againsuggested). In some implementations, some of the suggested selectedimages can be designated to be archived and some can be designated tonot be archived (e.g., particular selected images rejected for archivalby user input). If the decision is to archive, the method continues toblock 316. If not, the method continues to block 318.

In block 316, one or more selected images are archived. Archiving caninclude adding a respective archive attribute (e.g., label, which can beimage metadata) associated with the selected images to be archived, ormodifying a respective existing archival attribute (e.g., functionalimage identifier) associated with the selected images to indicate theimages are archived (e.g., setting a metadata flag to an archive state).In some implementations, archiving can include moving the image data ofthe archived images to be stored in an archive storage space of datastorage (e.g., a storage area, a storage device, a class of storage suchas long term storage, an archive folder, etc. that is different than thestorage used for non-archived images), e.g., to reduce storage spaceusage on user devices and/or server devices. In some implementations, animage database is updated to include an attribute “archived” for imagesthat have been archived.

In some implementations, images of the user's collection can be examinedfor archive status prior to or during display of one or more of theimages of the collection, e.g., the images are examined for the archivalattribute above. If the archival attribute is detected, the associated(archived) images are excluded from being displayed (e.g., notdisplayed) in views of non-archived images of the image collection. Forexample, one or more (non-archived) images of the collection other thanthe archived images are displayed in such views. In someimplementations, archived images may be separately viewable or may beviewable with non-archived images under particular conditions (e.g.,when a user provides input to select to view archived images). Themethod continues to block 318.

In block 318, in some implementations, archiving action feedback can beprovided to an image archiving model. For example, user decisions toarchive or not archive images, as well as determined image content,features, categories, and/or other image characteristics of archivedimages and non-archived images suggested for archival, can be providedto a machine-learning model to help train or tune the model.

In some implementations, with user permission, the user's selectionswhether to archive or to not archive images (e.g., in response torecommendations of block 312 and as obtained to perform block 314, or ifthe user manually archives images) are stored and used to adjust thearchiving process described herein, e.g., adjust one or more thresholdsor other parameters, add images to a whitelist or blacklist, etc. Forexample, the frequency or percentage of particular labels and/orcategories of images that are selected by the user to be archived can bedetermined, and/or the frequency or percentage of particular labelsand/or categories of images that are selected by the user to not bearchived. (The difference of such archiving and non-archiving selectionscan also be used.) In some examples, if the percentage of archivedimages having a particular label or category is higher than a thresholdor higher than an average percentage, then threshold(s) associated withthat label or category can be reduced to allow a greater number ofimages with that label or category to be suggested for archiving. Insome implementations, a machine-learning model can be trained with suchuser selections, or other model (e.g., statistical model, etc.) can usesuch frequencies of archiving and non-archiving user selections todetermine images to archive using the model (e.g., images having labelswith a high frequency of archiving).

In FIG. 3, various blocks (e.g., blocks 302-318) are illustrated asbeing performed sequentially. It will be appreciated however that theseblocks may be re-arranged as convenient to suit particular embodimentsand that these blocks or portions thereof may be performed concurrentlyin some embodiments. It will also be appreciated that in some examplesvarious blocks may be eliminated, divided into additional blocks, and/orcombined with other blocks. The table can then be used to determinethresholds based on values in the table.

FIG. 4 is a block diagram of an example device 400 which may be used toimplement one or more features described herein. In one example, device400 may be used to implement a client device, e.g., any of clientdevices 120-126 shown in FIG. 1. Alternatively, device 400 can implementa server device, e.g., server device 104, etc. In some implementations,device 400 may be used to implement a client device, a server device, ora combination of the above. Device 400 can be any suitable computersystem, server, or other electronic or hardware device as describedabove.

One or more methods described herein (e.g., 200 and/or 300) can be runin a standalone program that can be executed on any type of computingdevice, a program run on a web browser, a mobile application (“app”) runon a mobile computing device (e.g., cell phone, smart phone, tabletcomputer, wearable device (wristwatch, armband, jewelry, headwear,virtual reality goggles or glasses, augmented reality goggles orglasses, head mounted display, etc.), laptop computer, etc.).

In one example, a client/server architecture can be used, e.g., a mobilecomputing device (as a client device) sends user input data to a serverdevice and receives from the server the final output data for output(e.g., for display). In another example, all computations can beperformed within the mobile app (and/or other apps) on the mobilecomputing device. In another example, computations can be split betweenthe mobile computing device and one or more server devices.

In some implementations, device 400 includes a processor 402, a memory404, and I/O interface 406. Processor 402 can be one or more processorsand/or processing circuits to execute program code and control basicoperations of the device 400. A “processor” includes any suitablehardware system, mechanism or component that processes data, signals orother information. A processor may include a system with ageneral-purpose central processing unit (CPU) with one or more cores(e.g., in a single-core, dual-core, or multi-core configuration),multiple processing units (e.g., in a multiprocessor configuration), agraphics processing unit (GPU), a field-programmable gate array (FPGA),an application-specific integrated circuit (ASIC), a complexprogrammable logic device (CPLD), dedicated circuitry for achievingfunctionality, a special-purpose processor to implement neural networkmodel-based processing, neural circuits, processors optimized for matrixcomputations (e.g., matrix multiplication), or other systems.

In some implementations, processor 402 may include one or moreco-processors that implement neural-network processing. In someimplementations, processor 402 may be a processor that processes data toproduce probabilistic output, e.g., the output produced by processor 402may be imprecise or may be accurate within a range from an expectedoutput. Processing need not be limited to a particular geographiclocation, or have temporal limitations. For example, a processor mayperform its functions in “real-time,” “offline,” in a “batch mode,” etc.Portions of processing may be performed at different times and atdifferent locations, by different (or the same) processing systems. Acomputer may be any processor in communication with a memory.

Memory 404 is typically provided in device 400 for access by theprocessor 402, and may be any suitable processor-readable storagemedium, such as random access memory (RAM), read-only memory (ROM),Electrically Erasable Read-only Memory (EEPROM), Flash memory, etc.,suitable for storing instructions for execution by the processor, andlocated separate from processor 402 and/or integrated therewith. Memory404 can store software operating on the server device 400 by theprocessor 402, including an operating system 408, machine-learningapplication 430, other applications 412, and application data 414. Otherapplications 412 may include applications such as a data display engine,web hosting engine, image display engine, notification engine, socialnetworking engine, etc. In some implementations, the machine-learningapplication 430 and other applications 412 can each include instructionsthat enable processor 402 to perform functions described herein, e.g.,some or all of the methods of FIGS. 2 and/or 3.

The machine-learning application 430 can include one or morenamed-entity recognition (NER) implementations for which supervisedand/or unsupervised learning can be used. The machine learning modelscan include multi-task learning based models, residual taskbidirectional LSTM (long short-term memory) with conditional randomfields, statistical NER, etc. Other applications 412 can include, e.g.,functional image detection, functional image archiving suggestion,automatic functional image archiving, etc. One or more methods disclosedherein can operate in several environments and platforms, e.g., as astand-alone computer program that can run on any type of computingdevice, as a web application having web pages, as a mobile application(“app”) run on a mobile computing device, etc.

In various implementations, machine-learning application 430 may utilizeBayesian classifiers, support vector machines, neural networks, or otherlearning techniques. In some implementations, machine-learningapplication 430 may include a trained model 434, an inference engine436, and data 432. In some implementations, data 432 may includetraining data, e.g., data used to generate trained model 434. Forexample, training data may include any type of data suitable fortraining a model for functional image detection and archiving, such asimages, labels, thresholds, etc. Training data may be obtained from anysource, e.g., a data repository specifically marked for training, datafor which permission is provided for use as training data formachine-learning, etc. In implementations where one or more users permituse of their respective user data to train a machine-learning model,e.g., trained model 434, training data may include such user data. Inimplementations where users permit use of their respective user data,data 432 may include permitted data.

In some implementations, data 432 may include collected data such as mapdata, image data (e.g., satellite imagery, overhead imagery, etc.), gamedata, etc. In some implementations, training data may include syntheticdata generated for the purpose of training, such as data that is notbased on user input or activity in the context that is being trained,e.g., data generated from simulated conversations, computer-generatedimages, etc. In some implementations, machine-learning application 430excludes data 432. For example, in these implementations, the trainedmodel 434 may be generated, e.g., on a different device, and be providedas part of machine-learning application 430. In various implementations,the trained model 434 may be provided as a data file that includes amodel structure or form, and associated weights. Inference engine 436may read the data file for trained model 434 and implement a neuralnetwork with node connectivity, layers, and weights based on the modelstructure or form specified in trained model 434.

Machine-learning application 430 also includes a trained model 434. Insome implementations, the trained model 434 may include one or moremodel forms or structures. For example, model forms or structures caninclude any type of neural-network, such as a linear network, a deepneural network that implements a plurality of layers (e.g., “hiddenlayers” between an input layer and an output layer, with each layerbeing a linear network), a convolutional neural network (e.g., a networkthat splits or partitions input data into multiple parts or tiles,processes each tile separately using one or more neural-network layers,and aggregates the results from the processing of each tile), asequence-to-sequence neural network (e.g., a network that takes as inputsequential data, such as words in a sentence, frames in a video, etc.and produces as output a result sequence), etc.

The model form or structure may specify connectivity between variousnodes and organization of nodes into layers. For example, nodes of afirst layer (e.g., input layer) may receive data as input data 432 orapplication data 414. Such data can include, for example, images, e.g.,when the trained model is used for functional image archiving.Subsequent intermediate layers may receive as input output of nodes of aprevious layer per the connectivity specified in the model form orstructure. These layers may also be referred to as hidden layers. Afinal layer (e.g., output layer) produces an output of themachine-learning application. For example, the output may be a set oflabels for an image, an indication that an image is functional, etc.depending on the specific trained model. In some implementations, modelform or structure also specifies a number and/or type of nodes in eachlayer.

In different implementations, the trained model 434 can include aplurality of nodes, arranged into layers per the model structure orform. In some implementations, the nodes may be computational nodes withno memory, e.g., configured to process one unit of input to produce oneunit of output. Computation performed by a node may include, forexample, multiplying each of a plurality of node inputs by a weight,obtaining a weighted sum, and adjusting the weighted sum with a bias orintercept value to produce the node output.

In some implementations, the computation performed by a node may alsoinclude applying a step/activation function to the adjusted weightedsum. In some implementations, the step/activation function may be anonlinear function. In various implementations, such computation mayinclude operations such as matrix multiplication. In someimplementations, computations by the plurality of nodes may be performedin parallel, e.g., using multiple processors cores of a multicoreprocessor, using individual processing units of a GPU, orspecial-purpose neural circuitry. In some implementations, nodes mayinclude memory, e.g., may be able to store and use one or more earlierinputs in processing a subsequent input. For example, nodes with memorymay include long short-term memory (LSTM) nodes. LSTM nodes may use thememory to maintain “state” that permits the node to act like a finitestate machine (FSM). Models with such nodes may be useful in processingsequential data, e.g., words in a sentence or a paragraph, frames in avideo, speech or other audio, etc.

In some implementations, trained model 434 may include embeddings orweights for individual nodes. For example, a model may be initiated as aplurality of nodes organized into layers as specified by the model formor structure. At initialization, a respective weight may be applied to aconnection between each pair of nodes that are connected per the modelform, e.g., nodes in successive layers of the neural network. Forexample, the respective weights may be randomly assigned, or initializedto default values. The model may then be trained, e.g., using data 432,to produce a result.

For example, training may include applying supervised learningtechniques. In supervised learning, the training data can include aplurality of inputs (e.g., a set of images) and a corresponding expectedoutput for each input (e.g., one or more labels for each image). Basedon a comparison of the output of the model with the expected output,values of the weights are automatically adjusted, e.g., in a manner thatincreases a probability that the model produces the expected output whenprovided similar input.

In some implementations, training may include applying unsupervisedlearning techniques. In unsupervised learning, only input data may beprovided and the model may be trained to differentiate data, e.g., tocluster input data into a plurality of groups, where each group includesinput data that are similar in some manner. For example, the model maybe trained to identify image labels that are associated with functionalimages and/or select thresholds for functional image archivingrecommendation.

In another example, a model trained using unsupervised learning maycluster words based on the use of the words in data sources. In someimplementations, unsupervised learning may be used to produce knowledgerepresentations, e.g., that may be used by machine-learning application430. In various implementations, a trained model includes a set ofweights, or embeddings, corresponding to the model structure. Inimplementations where data 432 is omitted, machine-learning application430 may include trained model 434 that is based on prior training, e.g.,by a developer of the machine-learning application 430, by athird-party, etc. In some implementations, trained model 434 may includea set of weights that are fixed, e.g., downloaded from a server thatprovides the weights.

Machine-learning application 430 also includes an inference engine 436.Inference engine 436 is configured to apply the trained model 434 todata, such as application data 414, to provide an inference. In someimplementations, inference engine 436 may include software code to beexecuted by processor 402. In some implementations, inference engine 436may specify circuit configuration (e.g., for a programmable processor,for a field programmable gate array (FPGA), etc.) enabling processor 402to apply the trained model. In some implementations, inference engine436 may include software instructions, hardware instructions, or acombination. In some implementations, inference engine 436 may offer anapplication programming interface (API) that can be used by operatingsystem 408 and/or other applications 412 to invoke inference engine 436,e.g., to apply trained model 434 to application data 414 to generate aninference.

Machine-learning application 430 may provide several technicaladvantages. For example, when trained model 434 is generated based onunsupervised learning, trained model 434 can be applied by inferenceengine 436 to produce knowledge representations (e.g., numericrepresentations) from input data, e.g., application data 414. Forexample, a model trained for functional image archiving may producefunctional image labels and confidences for an image, a model trainedfor suggesting functional image archiving may produce a suggestion forone or more functional images to be archived, or a model for automaticarchiving may automatically archive certain functional images based onimage type, etc. In some implementations, such representations may behelpful to reduce processing cost (e.g., computational cost, memoryusage, etc.) to generate an output (e.g., a label, a classification, asentence descriptive of the image, etc.). In some implementations, suchrepresentations may be provided as input to a different machine-learningapplication that produces output from the output of inference engine436.

In some implementations, knowledge representations generated bymachine-learning application 430 may be provided to a different devicethat conducts further processing, e.g., over a network. In suchimplementations, providing the knowledge representations rather than theimages may provide a technical benefit, e.g., enable faster datatransmission with reduced cost. In another example, a model trained forfunctional image archiving may produce a functional image signal for oneor more images being processed by the model.

In some implementations, machine-learning application 430 may beimplemented in an offline manner. In these implementations, trainedmodel 434 may be generated in a first stage, and provided as part ofmachine-learning application 430. In some implementations,machine-learning application 430 may be implemented in an online manner.For example, in such implementations, an application that invokesmachine-learning application 430 (e.g., operating system 408, one ormore of other applications 412) may utilize an inference produced bymachine-learning application 430, e.g., provide the inference to a user,and may generate system logs (e.g., if permitted by the user, an actiontaken by the user based on the inference; or if utilized as input forfurther processing, a result of the further processing). System logs maybe produced periodically, e.g., hourly, monthly, quarterly, etc. and maybe used, with user permission, to update trained model 434, e.g., toupdate embeddings for trained model 434.

In some implementations, machine-learning application 430 may beimplemented in a manner that can adapt to particular configuration ofdevice 400 on which the machine-learning application 430 is executed.For example, machine-learning application 430 may determine acomputational graph that utilizes available computational resources,e.g., processor 402. For example, if machine-learning application 430 isimplemented as a distributed application on multiple devices,machine-learning application 430 may determine computations to becarried out on individual devices in a manner that optimizescomputation. In another example, machine-learning application 430 maydetermine that processor 402 includes a GPU with a particular number ofGPU cores (e.g., 1000) and implement the inference engine accordingly(e.g., as 1000 individual processes or threads).

In some implementations, machine-learning application 430 may implementan ensemble of trained models. For example, trained model 434 mayinclude a plurality of trained models that are each applicable to sameinput data. In these implementations, machine-learning application 430may choose a particular trained model, e.g., based on availablecomputational resources, success rate with prior inferences, etc. Insome implementations, machine-learning application 430 may executeinference engine 436 such that a plurality of trained models is applied.In these implementations, machine-learning application 430 may combineoutputs from applying individual models, e.g., using a voting-techniquethat scores individual outputs from applying each trained model, or bychoosing one or more particular outputs. Further, in theseimplementations, machine-learning application may apply a time thresholdfor applying individual trained models (e.g., 0.5 ms) and utilize onlythose individual outputs that are available within the time threshold.Outputs that are not received within the time threshold may not beutilized, e.g., discarded. For example, such approaches may be suitablewhen there is a time limit specified while invoking the machine-learningapplication, e.g., by operating system 408 or one or more otherapplications 412.

In different implementations, machine-learning application 430 canproduce different types of outputs. For example, machine-learningapplication 430 can provide representations or clusters (e.g., numericrepresentations of input data), labels (e.g., for input data thatincludes images, documents, etc.), phrases or sentences (e.g.,descriptive of an image or video, suitable for use as a response to aninput sentence, suitable for use to determine context during aconversation, etc.), images (e.g., generated by the machine-learningapplication in response to input), audio or video (e.g., in response aninput video, machine-learning application 430 may produce an outputvideo with a particular effect applied, e.g., rendered in a comic-bookor particular artist's style, when trained model 434 is trained usingtraining data from the comic book or particular artist, etc. In someimplementations, machine-learning application 430 may produce an outputbased on a format specified by an invoking application, e.g. operatingsystem 408 or one or more other applications 412. In someimplementations, an invoking application may be another machine-learningapplication. For example, such configurations may be used in generativeadversarial networks, where an invoking machine-learning application istrained using output from machine-learning application 430 andvice-versa.

Any of software in memory 404 can alternatively be stored on any othersuitable storage location or computer-readable medium. In addition,memory 404 (and/or other connected storage device(s)) can store one ormore messages, one or more taxonomies, electronic encyclopedia,dictionaries, thesauruses, knowledge bases, message data, grammars, userpreferences, and/or other instructions and data used in the featuresdescribed herein. Memory 404 and any other type of storage (magneticdisk, optical disk, magnetic tape, or other tangible media) can beconsidered “storage” or “storage devices.”

I/O interface 406 can provide functions to enable interfacing the serverdevice 400 with other systems and devices. Interfaced devices can beincluded as part of the device 400 or can be separate and communicatewith the device 400. For example, network communication devices, storagedevices (e.g., memory and/or database 106), and input/output devices cancommunicate via I/O interface 406. In some implementations, the I/Ointerface can connect to interface devices such as input devices(keyboard, pointing device, touchscreen, microphone, camera, scanner,sensors, etc.) and/or output devices (display devices, speaker devices,printers, motors, etc.).

Some examples of interfaced devices that can connect to I/O interface406 can include one or more display devices 420 and one or more datastores 438 (as discussed above). The display devices 420 that can beused to display content, e.g., a user interface of an output applicationas described herein. Display device 420 can be connected to device 400via local connections (e.g., display bus) and/or via networkedconnections and can be any suitable display device. Display device 420can include any suitable display device such as an LCD, LED, or plasmadisplay screen, CRT, television, monitor, touchscreen, 3-D displayscreen, or other visual display device. For example, display device 420can be a flat display screen provided on a mobile device, multipledisplay screens provided in a goggles or headset device, or a monitorscreen for a computer device.

The I/O interface 406 can interface to other input and output devices.Some examples include one or more cameras which can capture images. Someimplementations can provide a microphone for capturing sound (e.g., as apart of captured images, voice commands, etc.), audio speaker devicesfor outputting sound, or other input and output devices.

For ease of illustration, FIG. 4 shows one block for each of processor402, memory 404, I/O interface 406, and software blocks 408, 412, and430. These blocks may represent one or more processors or processingcircuitries, operating systems, memories, I/O interfaces, applications,and/or software modules. In other implementations, device 400 may nothave all of the components shown and/or may have other elementsincluding other types of elements instead of, or in addition to, thoseshown herein. While some components are described as performing blocksand operations as described in some implementations herein, any suitablecomponent or combination of components of environment 100, device 400,similar systems, or any suitable processor or processors associated withsuch a system, may perform the blocks and operations described.

In some implementations, logistic regression can be used forpersonalization (e.g., personalizing functional image archivingsuggestions based on a user's pattern of archiving activity). In someimplementations, the prediction model can be handcrafted including handselected functional labels and thresholds. The mapping (or calibration)from image content analysis space to a predicted precision within thefunctional image archiving space can be performed using a piecewiselinear model.

In some implementations, the functional image archiving system couldinclude a machine-learning model (as described herein) for tuning thesystem (e.g., selecting functional image labels and correspondingthresholds) to potentially provide improved accuracy. Inputs to themachine learning model can include image content analysis labels, and/oran image descriptor vector that describes appearance and includessemantic information about an image, where two images with similarfeature vectors tend to look the same. Example machine-learning modelinput can include labels for a simple implementation and can beaugmented with image descriptor vector features for a more advancedimplementation. Output of the machine-learning module can include aprediction of how likely a user is to want to archive an image should asuggestion to archive the image be surfaced.

FIG. 5 is a diagrammatic illustration of an example of user interface500 to provide a functional image archiving suggestion for one or moreimages, according to some implementations. User interface 500 caninclude a suggestion “card” or window that includes a display ofthumbnail images (502, 504, 506, 508, and 510) and an element (512)representing suggested images not shown in the thumbnails. For example,images 502-510 may be thumbnail images corresponding to a plurality ofimages that are identified as functional images, where a functionalimage is determined as described herein. For example, thumbnail images502-510 can be lower-resolution versions of higher-resolution images.While FIG. 5 shows five images 502-510, in various implementations, anynumber of images may be included in user interface 500. In someimplementations, e.g., implementations that do not include an imagepreview, thumbnail images 502-510 may not be included in user interface500. In some implementations, information about the images, e.g.,content labels (e.g., describing people or objects depicted), timestamp,location, etc. may be displayed in addition to or alternative tothumbnails 502-510.

User interface 500 further includes text 516 (“Archive 23 Photos?”, inthis example) In some implementations, one or more of the images (e.g.,images 502-510) may be top ranked functional image(s) (e.g., asdescribed above with reference to FIGS. 2 and 3) and one or moreportions of text 516 may be based on the identified functional images.For example, text 516 can include a description of one or morefunctional image categories in which all (or a portion of) the suggestedimages are members, and/or a type of image that is suggested to bearchived. In some examples for functional image categories, text 516 caninclude, “Archive these receipts?”, “Archive these documents?”, etc. Insome implementations, when users consent to use of image data, text 516may include one or more additional portions that are based on image dataassociated with one or more of the plurality of images (e.g., images502-510), such as image metadata. For example, the additional portionscan indicate location of capture, timestamp, etc. For example, text 516can include a date of capture or related description to date of capture(e.g., “from yesterday”). In some implementations, text 516 may includedefault text, e.g., “Archive n images?”, and not include portions basedon the plurality of images.

The user interface 500 can be receptive to user input that indicates tomaintain non-archived status, or to unarchive, the suggested images. Insome examples on a client device, handling user actions with respect tothe user interface 500 (e.g., archive suggestion card) can includereceiving user taps or other input on a dismiss button (not shown) orreceiving a touch gesture (e.g., swipe right) to dismiss, and respondingby removing the user interface 500 from the display and marking that setof images (e.g., that suggestion card) internally to not be displayedagain in further suggestions to archive images.

In some implementations, after a user taps (or selects) an image or the“Review and Archive” button 514 to reveal review page containing thefirst set of archive suggestions, the system can iterate through thesuggested images, e.g., display one suggested image at a time in asequence. When a user taps on the “More suggestions” button 512, thesystem can show one or more of the remaining images suggested forarchiving that are not displayed in FIG. 5 (e.g., display an amount ofimages appropriate for the display format, size of interface, etc.).

In some examples, when a user selects to archive the suggested imagesindicated in the user interface 500 (e.g., taps a displayed archivebutton or inputs a particular gesture on a touchscreen), the systemdisplays that the images in the set in user interface 500 are archivedand the suggestion interface 500 is dismissed and removed from thedisplay. When a user makes a selection of one or more suggested imagesin the user interface 500 and selects to archive the selected images(e.g., taps a displayed archive button or inputs a particular gesture ona touchscreen), the system displays that the selected images arearchived and the suggestion interface 500 is dismissed and removed fromthe display.

Image archiving suggestion cards can be populated with suggested imagesfor archiving in response to the client requesting the card from anotification system, such that the notification system renders the card.In some implementations, before a user provides user input to select theimage archiving suggestion card to reveal the suggestions in the reviewpage, one or more images may have been deleted from the image collectionor archived, or new suggestions may have been inserted via an upload bythe user. Rather than modifying various handlers to directly edit thecard whenever suggestions appear and disappear, a functional imageidentifier associated with one or more image(s) can be modified instead(e.g., where the functional image identifier can include functionalimage category, image type, and/or confidence score value). Uponrefreshing the user interface, the archiving suggestion card's new setof suggestions is also modified based on the updated functional imageidentifier.

Providing an identifier specifying functional image category, type,and/or confidence score value for an image enables a functional imagearchiving suggestion system to fetch images by threshold confidencescore without a need to create other data structures, which may belimited in total number that can exist at one time in someimplementations. In some implementations, a functional image archivingsystem can tune the threshold confidence score for archive suggestionsto permit adjustments to be made without requiring a retroactive processto reassign images as suggestions.

In some implementations, images suggested for archiving can include anassociated suggestion state. The suggestion state can include states ofpending, accepted, or rejected. Suggestion state can transition frompending to either accepted or rejected when a user takes action on thesuggested image (e.g., accept or reject archival status for thesuggested image). In some implementations, when pending suggestions areacted upon by the user, the functional image identifier of the image canbe deleted. In some examples, if a user archives the image in any way,the suggestion to archive for the image is considered accepted. If theuser does not archive an image per suggestion from an assistant card andsubsequent review page, the suggestion to archive for the image can beconsidered rejected.

Some implementations can include automatic unarchiving of an archivedimage based one or more parameters or events, e.g., in response tounarchiving criteria being met. Unarchiving criteria for an archivedimage can include a period of time having elapsed since the image wasarchived, the image being included in one or more particular functionalimage categories, the image depicting one or more particular contentfeatures or types of content features (e.g. receipts, etc.), the imagehaving a functional image score below a particular threshold, etc.

In some implementations, unarchiving an image can include displaying orsurfacing one or more archived images as a notification, e.g., on adisplayed card or highlighting the images in a displayed view of images.In some implementations, a prompt for user input can be displayed withthe notification, requesting the user to accept or reject unarchiving ofany of the archived image(s).

In an example of an image archiving system performing an unarchiveoperation, an archived image of a receipt may be surfaced (e.g.,displayed) within a graphical user interface after a given time period(e.g., 2 months, one year, etc.) as a reminder to extend a warranty,return an item, etc. In another example, an archived photo of a concertadvertisement may be surfaced when a user next accesses a ticketpurchase app or web site. In yet another example, an archived photo of aproduct (e.g., “bottle of wine”) can be surfaced when the user anddevice is next located in a store that sells that type of product, etc.In still another example, an archived video of a birthday celebrationcan be surfaced the following year on the same date it was captured theprevious year. In still another example, an archived video of hurricanedamage to a structure can be surfaced when a user is detected to bewriting an email to the insurer of the structure, as detected by thesystem if user consent has been obtained.

In some implementations, a functional image archiving system can have adelay period (e.g., 7 days) prior to adding a functional image to asuggestion interface such as the archiving suggestion card. The delayperiod can be measured from the time of accessibility (e.g., upload orcapture) of the image. In some implementations, a list of suggestedimages to archive can be stored and/or the dates/times when thosesuggested images are to be surfaced for the user can be stored. In someexamples, the list of suggested images can include a particular amountof images (e.g., up to 120 images in descending functional image scoreorder). In some implementations, functional image archive suggestioncards can be presented once per a given time period, e.g., a given timeperiod is provided between display of archive suggestion cards (e.g.,once every 20 days if the user has uploaded an image that has an archivesuggestion within that time period). The time period can be specified bythe user in some implementations.

In some implementations, the given time period between suggestion cardscan be adjusted based on the user's acceptance or rejection (dismissal)of one or more prior archiving suggestions. For example, if a userdismisses (e.g., rejects or delays acting on) a functional image archivesuggestion card that was displayed after a first time period (e.g., 20days) since the previous suggestion card, then the next functional imagearchive suggestion card can be designated to show up after a second,longer time period (e.g., 30 days) than the first time period. If a useraccepts one or more suggestions to archive a functional image, the nextfunctional image archive suggestion card may be surfaced sooner (e.g.,after a time period of 5 days) than the previous time period betweenpresentation of functional image archive suggestion cards.

Some implementations can be implemented as part of a hardware system(e.g., servers, scanners, cameras, etc.) and/or as part of a softwaresystem (e.g., image management application, photo application, datastorage application, image recognition application, assistive cameraapplication, etc.). Some implementations can use the signals and scoresdescribed herein (e.g., functional image signal, functional image score,etc.) to determine a display status of images in any display context,e.g., when displaying images in any interface or application, e.g.,without performing archive actions or providing archive options to theuser. For example, a system can use these signals and scores todetermine that particular functional images should not be displayed withother images.

One or more methods described herein (e.g., method 200 or 300) can beimplemented by computer program instructions or code, which can beexecuted on a computer. For example, the code can be implemented by oneor more digital processors (e.g., microprocessors or other processingcircuitry), and can be stored on a computer program product including anon-transitory computer readable medium (e.g., storage medium), e.g., amagnetic, optical, electromagnetic, or semiconductor storage medium,including semiconductor or solid state memory, magnetic tape, aremovable computer diskette, a random access memory (RAM), a read-onlymemory (ROM), flash memory, a rigid magnetic disk, an optical disk, asolid-state memory drive, etc. The program instructions can also becontained in, and provided as, an electronic signal, for example in theform of software as a service (SaaS) delivered from a server (e.g., adistributed system and/or a cloud computing system). Alternatively, oneor more methods can be implemented in hardware (logic gates, etc.), orin a combination of hardware and software. Example hardware can beprogrammable processors (e.g. Field-Programmable Gate Array (FPGA),Complex Programmable Logic Device), general purpose processors, graphicsprocessors, Application Specific Integrated Circuits (ASICs), and thelike. One or more methods can be performed as part of or component of anapplication running on the system, or as an application or softwarerunning in conjunction with other applications and operating system.

One or more methods described herein can be run in a standalone programthat can be run on any type of computing device, a program run on a webbrowser, a mobile application (“app”) run on a mobile computing device(e.g., cell phone, smart phone, tablet computer, wearable device(wristwatch, armband, jewelry, headwear, goggles, glasses, etc.), laptopcomputer, etc.). In one example, a client/server architecture can beused, e.g., a mobile computing device (as a client device) sends userinput data to a server device and receives from the server the finaloutput data for output (e.g., for display). In another example, allcomputations can be performed within the mobile app (and/or other apps)on the mobile computing device. In another example, computations can besplit between the mobile computing device and one or more serverdevices.

Although the description has been described with respect to particularimplementations thereof, these particular implementations are merelyillustrative, and not restrictive. Concepts illustrated in the examplesmay be applied to other examples and implementations.

Due to the nature of generating a functional image archiving suggestionbased on analysis of images, implementations discussed herein mayrequire access to user data such as images and current, historical orfuture archiving actions of relevant users. In situations in whichcertain implementations discussed herein may collect or use personalinformation about users (e.g., user data, user image data, image sharingdata, information about a user's social network, user's location andtime, user's biometric information, user's activities and demographicinformation), users are provided with one or more opportunities tocontrol whether the personal information is collected, whether thepersonal information is stored, whether the personal information isused, and how the information is collected about the user, stored andused. That is, the systems and methods discussed herein collect, storeand/or use user personal information specifically upon receivingexplicit authorization from the relevant users to do so. In addition,certain data may be treated in one or more ways before it is stored orused so that personally identifiable information is removed (e.g., thesharing suggestion system may anonymously identify important people byfeatures other than personally identifiable information such as name oruser name). As one example, a user's identity may be treated so that nopersonally identifiable information can be determined. As anotherexample, a user's geographic location or a location associated with userimages may be generalized to a larger region so that the user'sparticular location cannot be determined.

Note that the functional blocks, operations, features, methods, devices,and systems described in the present disclosure may be integrated ordivided into different combinations of systems, devices, and functionalblocks as would be known to those skilled in the art. Any suitableprogramming language and programming techniques may be used to implementthe routines of particular implementations. Different programmingtechniques may be employed, e.g., procedural or object-oriented. Theroutines may execute on a single processing device or multipleprocessors. Although the steps, operations, or computations may bepresented in a specific order, the order may be changed in differentparticular implementations. In some implementations, multiple steps oroperations shown as sequential in this specification may be performed atthe same time.

We claim:
 1. A computer-implemented method comprising: programmaticallyanalyzing a plurality of images to detect respective content in each ofthe plurality of images; identifying one or more labels for each imagebased on the content of the image, wherein each label for the image isassociated with a corresponding confidence score; identifying a subsetof the plurality of images that each have at least one label that is afunctional label, wherein the corresponding confidence score associatedwith each of the at least one label meets a threshold; and causing atleast one image of the subset of the plurality of images to be archivedby associating an archive attribute with the at least one image suchthat the at least one image is excluded from automatically generatedcreations based on the plurality of images.
 2. The computer-implementedmethod of claim 1, wherein the automatically generated creations includeimages of the plurality of images and are at least one of: a photo book,an image animation, a slide show, or a collage.
 3. Thecomputer-implemented method of claim 1, further comprising receiving arequest to determine images with functional labels in the plurality ofimages, the request provided by a system that is to automaticallygenerate a creation based on the plurality of images, whereinidentifying the subset of the plurality of images and causing the atleast one image to be archived is performed in response to the request.4. The computer-implemented method of claim 1, further comprising:determining a functional image score for each image of the plurality ofimages; sorting the images in the subset according to the functionalimage score; and selecting a number of images of the plurality of imageshaving top functional images scores to be archived.
 5. Thecomputer-implemented method of claim 1, further comprising: maintaininga blacklist including particular content features that, if detected inan image, automatically disqualify the image from being in the subset;or maintaining a whitelist including particular content features that,if detected in an image, automatically qualify the image to be in thesubset, wherein identifying the subset of the plurality of imagesincludes qualifying or disqualifying one or more images of the pluralityof images based on the blacklist or the whitelist.
 6. Thecomputer-implemented method of claim 1, further comprising: mapping theone or more corresponding confidence scores from an image contentanalysis space in which respective objects are identified in theplurality of images, to a functional image archiving space in which theplurality of images are evaluated for archival, wherein a probability ofa respective object being identified in a respective image is differentfrom a probability of archiving the respective image that includes therespective object.
 7. The computer-implemented method of claim 1,further comprising: identifying one or more archival categories in whicheach respective image of the at least one image in the subset isincluded, wherein each archival category is associated with acorresponding category confidence score threshold, wherein causing eachrespective image of the at least one image of the subset to be archivedis in response to determining that the respective image has a categoryconfidence score that meets the category confidence score threshold forat least one of the one or more archival categories in which therespective image is included.
 8. The computer-implemented method ofclaim 7, wherein the one or more archival categories include at leastone of: receipt, document, newspaper, note, product label, screenshot,restaurant menu, identification, or business card.
 9. Thecomputer-implemented method of claim 7, further comprising: generating arecommendation to archive the at least one image of the subset, whereinthe recommendation includes at least one archival category of the one ormore archival categories in which the at least one image is included;and causing output of the recommendation and the at least one archivalcategory by a device, wherein causing the at least one image of thesubset to be archived is based on user input received at the device inresponse to the recommendation.
 10. The computer-implemented method ofclaim 9, wherein causing output of the recommendation includes providinga user interface that displays the at least one image and includes asuggestion card or a suggestion chip element.
 11. Thecomputer-implemented method of claim 1, wherein: causing the at leastone image of the subset to be archived is performed automatically,without user input, in response to determination that at least oneconfidence score of the at least one image meets a first archivingthreshold; or causing the at least one image of the subset to bearchived is based on user input received at a device in response todetermination that the at least one confidence score of the at least oneimage meets a second archiving threshold and does not meet the firstarchiving threshold, wherein the user input is received in response tooutputting a recommendation by the device to archive the at least oneimage.
 12. The computer-implemented method of claim 1, furthercomprising automatically unarchiving the at least one image of thesubset in response to one or more unarchiving criteria being met,wherein the one or more unarchiving criteria include one or more of: aperiod of time having elapsed since the at least one image was archived,or the at least one image depicting one or more particular contentfeatures.
 13. The computer-implemented method of claim 1, furthercomprising programmatically analyzing metadata associated with theplurality of images to determine the one or more labels and to determinethe one or more corresponding confidence scores.
 14. Thecomputer-implemented method of claim 1, wherein the method is performedat one of: a time the plurality of images is obtained; or a time that isafter a time period occurring after the plurality of images wasobtained.
 15. A system comprising: a memory with instructions storedthereon; and at least one processor configured to access the memory andto execute the instructions, wherein the instructions cause theprocessor to perform operations comprising: programmatically analyzing aplurality of images to detect respective content in each of theplurality of images; identifying one or more labels for each image basedon the content of the image, wherein each label for the image isassociated with a corresponding confidence score; identifying a subsetof the plurality of images that each have at least one label that is afunctional label, wherein the corresponding confidence score associatedwith each of the at least one label meets a threshold; and causing atleast one image of the subset of the plurality of images to be archivedby associating an archive attribute with the at least one image suchthat the at least one image is excluded from automatically generatedcreations based on the plurality of images.
 16. The system of claim 15,wherein the automatically generated creations include images of theplurality of images and are at least one of: a photo book, an imageanimation, a slide show, or a collage.
 17. The system of claim 15,wherein: the operation of causing the at least one image of the subsetto be archived is performed automatically, without user input, inresponse to determination that at least one confidence score of the atleast one image meets a first archiving threshold; and the operation ofcausing the at least one image of the subset to be archived is based onuser input received at a device in response to determination that the atleast one confidence score of the at least one image meets a secondarchiving threshold and does not meet the first archiving threshold,wherein the user input is received in response to outputting arecommendation by the device to archive the at least one image.
 18. Anon-transitory computer readable medium having stored thereon softwareinstructions that, when executed by a processor, cause the processor toperform operations comprising: programmatically analyzing a plurality ofimages to detect respective content in each of the plurality of images;identifying one or more labels for each image based on the content ofthe image, wherein each label for the image is associated with acorresponding confidence score; identifying a subset of the plurality ofimages that each have at least one label that is a functional label,wherein the corresponding confidence score associated with each of theat least one label meets a threshold; and causing at least one image ofthe subset of the plurality of images to be archived by associating anarchive attribute with the at least one image such that the at least oneimage is excluded from automatically generated creations based on theplurality of images.
 19. The non-transitory computer readable medium ofclaim 18, wherein the operations further comprise: identifying one ormore archival categories in which each respective image of the at leastone image in the subset are included, wherein each archival category isassociated with a corresponding category confidence score threshold,wherein causing each respective image of the at least one image of thesubset to be archived is in response to determining that the respectiveimage has a category confidence score that meets the category confidencescore threshold for at least one of the one or more archival categoriesin which the respective image is included, and wherein the one or morearchival categories include at least one of: receipt, document,newspaper, note, product label, screenshot, restaurant menu,identification, or business card; generating a recommendation to archivethe at least one image of the subset, wherein the recommendationincludes at least one archival category of the one or more archivalcategories in which the at least one image is included; causing outputof the recommendation and the at least one archival category by adevice; and receiving user input at the device in response to therecommendation, wherein causing the at least one image of the subset tobe archived is based on the user input received at the device.
 20. Thenon-transitory computer readable medium of claim 18, wherein theoperations further comprise automatically unarchiving the at least oneimage of the subset in response to one or more unarchiving criteriabeing met, wherein the one or more unarchiving criteria include one ormore of: a period of time having elapsed since the at least one imagewas archived, the at least one image being included in one or moreparticular functional image categories, the at least one image depictingone or more particular content features, or the at least one imagehaving a confidence score below a particular threshold.